Certainty rating in pre-and post-tests of study modules in an online clinical pharmacy course - A pilot study to evaluate teaching and learning

BACKGROUND Graduate and post-graduate education for health professionals is increasingly delivered in an e-learning environment, where automated, continuous formative testing with integrated feedback can guide students' self-assessment and learning. Asking students to rate the certainty they assign to the correctness of their answers to test questions can potentially provide deeper insights into the success of teaching, with test results informing course designers whether learning outcomes have been achieved. It may also have implications for decision making in clinical practice. METHODS A study of pre-and post-tests for five study modules was designed to evaluate the teaching and learning within a pharmacotherapeutic course in an online postgraduate clinical pharmacy program. Certainty based marking of multiple choice questions (MCQ) was adapted for formative pre- and post-study module testing by asking students to rate their certainty of correctness of MCQ answers. Paired t-tests and a coding scheme were used to analyse changes in answers and certainty between pre-and post-tests. A survey evaluated students' experience with the novel formative testing design. RESULTS Twenty-nine pharmacists enrolled in the postgraduate program participated in the study. Overall 1315 matched pairs of MCQ answers and certainty ratings between pre- and post-module tests were available for evaluation. Most students identified correct answers in post-tests and increased their certainty compared to pre-tests. Evaluation of certainty ratings in addition to correctness of answers identified MCQs and topic areas for revision to course designers. A survey of students showed that assigning certainty ratings to their answers assisted in structuring and focusing their learning throughout online study modules, facilitating identification of areas of uncertainty and gaps in their clinical knowledge. CONCLUSIONS Adding certainty ratings to MCQ answers seems to engage students with formative testing and feedback and focus their learning in a web-based postgraduate pharmacy course. It also offers deeper insight into the successful delivery of online course content, identifying areas for improvement of teaching and content delivery as well as test question design.